(Not Applicable)
1. Field of the Invention
This invention relates to optical metrology, and particularly to the problem of making accurate non-contact dimensional measurements of objects that are viewed through an endoscope.
2. Description of Related Art
A. Endoscopic Measurements
Making accurate dimensional measurements of objects viewed through endoscopes is important to aerospace as well as other industries in which expensive equipment must undergo periodic internal inspections to maintain safe operation. Such measurements also have medical applications, where the internal condition of a patient is evaluated prior to or during surgery by viewing through an endoscope.
The fundamental problems in making an accurate measurement through an endoscope are that the magnification of the image varies rapidly with the range of the object, and that objects of interest (defects) lie on surfaces which are curved in three dimensions; thus the magnification varies from one point on the object to another. What is needed is a fully three-dimensional measurement, that is, one which determines the depth, as well as the height and width, of an object.
Endoscopes are long and narrow optical systems, typically circular in cross-section, which can be inserted through a small opening in an enclosure to give a view of the interior. They almost always include a source of illumination which is conducted along the interior of the scope from the outside (proximal) end to the inside (distal) end, so that the interior of the chamber can be viewed even if it contains no illumination. Endoscopes are divided into two basic types: these are the rigid xe2x80x9cborescopesxe2x80x9d and the flexible xe2x80x9cfiberscopesxe2x80x9d or xe2x80x9cvideoscopesxe2x80x9d.
Probably the simplest approach to obtaining quantitative object size information is to place a physical scale in contact with the object to be measured. U.S. Pat. No. 4,825,259 to Berry teaches this approach, as does Diener, in U.S. Pat. No. 5,803,680. Berry attaches the scale to the distal tip of an endoscope, while Diener attaches the scale to the distal end of a remote machining apparatus, where the apparatus includes an endoscope.
One problem with this approach is that it is sometimes not possible to insert the desired scale through the available access port. As an alternative, Watanabe in U.S. Pat. No. 4,721,098 teaches the construction and use of a measurement scale apparatus that has both a collapsed configuration and an expanded configuration, so that the scale can be passed through the access port when the apparatus is collapsed, and then the apparatus can be erected to an operating configuration once it is near the object of interest.
Other problems with such direct physical scale approaches are that the objects of interest are almost never flat and oriented in the correct plane so that the scale can lie against them, and that it is often not permissible to touch the objects of interest with a rigid scale. Even the seemingly simple task of bringing the scale into contact with the object so that the points of interest on the object are adjacent to the indicia of the scale is often difficult. In many cases, the precision with which the position of the scale can be manipulated is insufficient. In addition, it is difficult to determine that all of the desired indicia on the scale are in contact with the object rather than lying either in front of or behind the object.
Even when the object is suitable for measurement with a physical scale and the scale can be manipulated satisfactorily, there is an additional requirement to manipulate the end of the endoscope into the correct position to make the desired measurement accurately. Ideally, the scale is oriented perpendicular to the line of sight of the endoscope. If the scale is not so oriented, then determining when the points on the object are aligned with the indicia of the scale becomes more difficult and subject to error.
In the current marketplace, there are many existing so-called endoscopic xe2x80x9cmeasurementxe2x80x9d systems that simply make straightforward two-dimensional measurements based on applying a scale factor to the image viewed through the scope. Such systems are today almost always implemented with a video camera attached to the proximal end of the endoscope, and with a digital xe2x80x9cframe grabberxe2x80x9d being used to acquire and store the video images. The measurement is then made simply by counting xe2x80x9cpixelsxe2x80x9d between features of interest in the digital video images, and a computing device multiplies this number by an appropriate scale factor, where the scale factor may or may not take into account the Seidel distortion of the optical system.
Such systems are sometimes useful, but they are severely limited by the following requirements for the two dimensional measurement to have a meaningful relationship to the true dimension of the object. First, the object being measured has to be oriented at a known angle with respect to the line of sight. Second, the distance of the object from the endoscope optical system has to be known in order to determine the correct scale factor. The angle of the object can sometimes be estimated from the known geometrical relationship of the various components of the device being inspected. The magnification of the object is sometimes estimated by incorporating an auxiliary object of known size, such as a wire, to be compared to the object of interest, or the distance is estimated by adding a physical projection to the endoscope. An example of the latter is the system of Krauter, U.S. Pat. No. 5,047,848, in which a flexible distance gauging element is attached to the distal tip of an endoscope.
Clearly there are many sources of error in such xe2x80x9cmeasurementsxe2x80x9dxe2x80x94these are in fact crude estimates that are useful in only a limited set of circumstances. One cannot often depend on these for critical applications. Thus, much effort has gone into the development of non-contact, truly three dimensional, methods of measurement through endoscopes.
Many of the prior art approaches to three-dimensional, non-contact measurements involve adding optical projection apparatus to the distal tip of the endoscope. Besides having the problem that the measurement precision is poor, these approaches inherently involve adding apparatus to an area where space is already at a premium. The distal tip of the endoscope must be kept as small as possible in order to allow inspections in close quarters, and to allow the endoscope to be inserted through access ports which are as small as possible. Thus, ideally, one wants to be able to make the measurement without adding anything to the distal tip of an endoscope.
U.S. Pat. No. 4,895,431, xe2x80x9cMethod Of Processing Endoscopic Imagesxe2x80x9d, to Tsujiuchi, et. al., describes a number of methods to mathematically process two images obtained from different camera positions to derive object surface contour information. The two images are obtained by bending the end of a fiberscope. The bending is achieved using the internal articulation capability of the scope. Their technique assumes a simple linear relationship between the bend angle of the fiberscope and the offset of the nodal point of the optical system to estimate the geometrical relationship of the images. The image processing begins by correcting for distortion, then doing correlations over a series of smaller and smaller sub-images. The patent teaches that one can thereby derive full three-dimensional position data in the overlap region between the images.
One problem with this is that the bending of the end of a fiberscope is subject to a number of difficult-to-correct mechanical errors that make the feasibility of this approach questionable.
U.S. Pat. No. 5,432,543, xe2x80x9cEndoscopic Image Processing Device for Estimating Three-Dimensional Shape of Object Based on Detection of Same Point on a Plurality of Different Imagesxe2x80x9d, to Hasegawa, et. al., is an improvement to U.S. Pat. No. 4,895,431, which avoids the problem of mechanical errors. The new approach is to estimate the relative positions of the imaging optical system and the object in addition to the previous estimation of the 3D contour of the object, all by using sophisticated image processing. The resulting system is purported to be able to handle even the case of a moving object.
A first problem with the approach of both Tsujiuchi et. al., and Hasegawa, et. al., is that these systems require the use of a complicated and expensive image processing device. A second problem is that they are not directed toward obtaining simple measurements of a few dimensions on the object; instead they are directed toward determining the three dimensional shape of a surface. Thus, they do not calculate or even consider the coordinates of any individual point on the object; the processes are all based on correlations over sub-images. This inherently means that for any single point of interest on an object, the estimated position of that point depends on the characteristics of neighboring points as well as on its own characteristics.
To address the deficiencies of these and other three-dimensional, non-contact endoscopic measurement techniques, I invented the system described in U.S. Pat. No. 6,009,189, which is incorporated herein by reference. Much of the same information was also published as xe2x80x9cApparatus and method for making accurate three-dimensional size measurements of inaccessible objectsxe2x80x9d, International Pat. No. Publication Number WO 98/07001, World Intellectual Property Organization, Geneva, Feb. 19, 1998. The latter document was produced without the introduction of errors in the printing process.
In this previous system, an imaging camera is subjected to a precision motion from a first viewing position to a second viewing position, and measurements are made using the information contained in both views of the object. As explained in the referenced documents, this system is an improved version of a more general technique that I call perspective dimensional measurement.
The disclosure of U.S. Pat. No. 6,009,189 emphasizes the need for and methods of obtaining the best possible accuracy in the endoscopic measurement. Experimental results using this previous system were published in Proceedings of SPIE, vol. 3397, pp. 264-276, 1998. Using my teachings, it is now possible to make three-dimensional measurements to a precision of better than 1 part in 1000 of the range to the object using standard video endoscopic equipment when the object of interest has features with sharp, high-contrast edges. This is at least 10 to 20 times better than had been achievable with prior art three-dimensional endoscopic measurement systems.
While my previous system allows one to make accurate measurements with existing rigid borescopes, if that system is to be used with a flexible endoscope, it requires that a completely new flexible scope be designed and manufactured. In addition, the high level of measurement accuracy offered by that system is not always required, although one does usually want a better measurement than was provided by earlier systems.
B. Photogrammetric Measurements
In recent decades new applications of photogrammetry (i.e., the science and technology of making measurements from photographs) have been introduced under the names xe2x80x9cnon-topographic photogrammetryxe2x80x9d or xe2x80x9cclose-range photogrammetryxe2x80x9d. In the typical application of this art, a large structure such as a dam, a building, a ship, or an archeological site is accurately measured using a number of photographs taken from different viewing positions. Often, more than two viewing positions are used. Such measurements are miniature engineering projects, requiring extensive planning before the data are acquired and extensive data analysis after the data are acquired. The equipment required is complex and expensive, however the measurement accuracies obtainable are extremely high, with 1 part in 10,000 of the range being considered minimal accuracy and in excess of 1 part in 200,000 being achievable. A classic reference to this field is the book Handbook of Non-Topographic Photogrammetry, H. M. Karara, ed., American Society of Photogrammetry, 1979.
In one type of non-topographic photogrammetry a large number of alignment targets are attached to the object to be measured. The positions of these targets, when determined by the photogrammetric process, are used to represent the object. That is, in these measurements the points on the object to be measured must be preselected during the planning stage.
In another type of photogrammetric measurement, a so-called control frame is placed around the object of interest and the object and the control frame are photographed together. If the three-dimensional positions of a sufficient number of control points on the control frame have been independently determined, then desired points on the object can be measured. In this case, the points to be measured on the object need not be preselected. This avoids the extensive preplanning but it still requires the extensive (and expensive) post analysis of the photographs.
In still other applications of photogrammetry, the object of interest is considered to have a number of well defined points for which the three-dimensional positions are already known. These specific object points are then used as control for photogrammetric measurements of other portions of the object. Such a system for making measurements inside a gas turbine engine is proposed in an article by D. Whittaker, xe2x80x9cPhotogrammetry with Endoscopexe2x80x9d, International Archives of Photogrammetry and Remote Sensing, vol. XXX, Part 5, pp. 437-442; 1994. As described by Whittaker, setting up to make the proposed measurements inside the engine is a major engineering project; and after the scheme is set up, extensive data reduction would be necessary for any individual measurement to be made.
The problem with the application of photogrammetry to the endoscopic measurement application is the difference between the goals of photogrammetry and the goals of endoscopic measurement. In photogrammetry, the goal is usually to characterize the structure as a whole; thus a very large amount of data is desired. In endoscopic measurements we are concerned with discrete positions on the object of interest, typically sites of damage such as nicks, cracks, or pits in industrial applications, or the size of tumors or other features of the body in medical applications. Often a single dimension between two points on the object is the only information desired. Rather than needing the extremely high accuracy of which photogrammetry is capable, the precision of an endoscopic measurement needs to be between 1 part in several hundred and 1 part in 10,000 of the range. Rather than being an engineering project that takes hours, days or weeks, the endoscopic measurement must be performed in minutes. Rather than allowing for the use of expensive photoreduction hardware and software, the endoscopic measurement must be as inexpensive as possible. For these reasons, to my knowledge, photogrammetry per se has not found application in endoscopic measurements.
In fact, the differences between the practice of photogrammetry and the needs of endoscopic measurement are so profound that its practitioners have not considered photogrammetry to be applicable to endoscopic measurement at all. For instance, in the book Close-Range Photogrammetry and Surveying. State of the Art, American Society of Photogrammetry; 1985, in an article on applications in medicine, a major academic figure in photogrammetry, professor F. H. Moffit of the University of California at Berkeley, made the statement: xe2x80x9cVarious types of endoscopes such as the colonoscope and the sigmoidoscope can be used to photograph internal parts for later examination and interpretation; however, this is outside the purview of the photogrammetrist.xe2x80x9d (p. 761). A similar sentiment was recently stated by I. Newton and H. L. Mitchell in the Chapter entitled xe2x80x9cMedical Photogrammetryxe2x80x9d in the book Close Range Photogrammetry and Machine Vision, K. B. Atkinson, ed., Whittles Publishing, Bristol, England, 1996. They state: xe2x80x9cAlthough photogrammetry, along with other optical techniques, appears to have the disadvantage that it is associated with the exterior of the body, rather than the interior which is often seen as so relevant to health, it is found in practice that external studies are valuable in many situations for a number of reasons.xe2x80x9d (p. 304).
C. Summary of Prior Art
To summarize the deficiencies of the prior art in endoscopic measurements, methods requiring physical contact for dimensional measurements are not often applicable in practice. Many available endoscopic non-contact measurement systems are not truly three-dimensional, and therefore are inaccurate. Optical projection approaches involve increasing the size of the distal tip of the endoscope , they require that a specialized measurement endoscope be acquired at high cost, and their accuracy is insufficient. Image processing approaches are complicated and expensive, are not directed toward obtaining measurements of the distances between a few specific points on an object, and their accuracies are currently unknown. My previous perspective dimensional measurement system provides three-dimensional measurements of nigh accuracy, but it cannot be used with standard, unmodified flexible endoscopes. The practice of close-range photogrammetry has been directed toward making measurements of extremely high accuracy where the time necessary to make the measurement, and the cost of the equipment required are completely out of the question for the endoscopic measurement application.
What is needed is an inexpensive system that can be used with existing flexible endoscopes that provides a non-contact, true three-dimensional measurement. The system should not involve adding any apparatus to the distal tip of an endoscope, even as an add-on accessory. In addition, it would be ideal if this new system could be easily retrofitted to existing two-dimensional pixel counting endoscopic xe2x80x9cmeasurementxe2x80x9d systems to upgrade their accuracy and usefulness.
The present invention resolves the problems identified with the prior art and offers additional advantages as well. It is a first object of this invention to provide a non-contact method for making true three-dimensional measurements of the distances between individual points on an object that can be used with any endoscope, without requiring any modification of the endoscope.
A second object of this invention is to provide a non-contact method for making three-dimensional measurements which does not require the use of an image processing device.
A third object is to provide a non-contact method and apparatus for measuring three-dimensional distances which are not fully contained in any single view of a measurement camera.
A fourth object is to provide a method for making non-contact three-dimensional measurements which does not require any pre-calibration of the measurement endoscope.
A fifth object is to provide a variety of apparatus for implementing the methods that enable accurate three-dimensional measurements to be made using any endoscope.
A sixth object is to provide a variety of measurement apparatus that enable three-dimensional measurements to be made in a wide variety of applications where the object to be measured is inaccessible except through access openings in an enclosure.
A seventh object is to provide a variety of measurement apparatus which can be inserted through existing inspection ports in equipment to be inspected.
An eighth object is to provide a method for determining three-dimensional coordinates for at least one point on an object in which the method does not require that an endoscope be subjected to an accurately predetermined motion.
The methods of the instant invention are adaptations of my improved perspective dimensional measurement technology in which the motion of the camera, rather than being precisely controlled, is instead determined by using additional information contained within images obtained at the two viewing positions. This additional information is added to the scene being viewed by placing an array of reference target points into the scene next to, and fixed with respect to, the object of interest.
Rather than simply measuring the positions of the images of object points of interest in each of the two views, in the new methods one also measures the positions of the images of a number of the reference target points. These additional data, together with predetermined spatial relationships between the reference target points and together with internal calibration parameters of the camera, enable one to determine the positions and orientations of the measurement camera in three-dimensional space for each of the views. Optionally, one may either predetermine the internal calibration parameters of the camera, or determine those parameters simultaneously with the determination of the positions and orientations of the camera. The latter option requires one to measure the positions of a larger number of reference target points than does the former.
Once the locations and orientations of the measurement camera have been determined for each of the views, one can determine three-dimensional distances between the points of interest on the object using the improved perspective dimensional measurement technique taught in my previous application, now U.S. Pat. No. 6,009,189. Measurements can be made with the new methods with either the xe2x80x9cmode 1xe2x80x9d or xe2x80x9cmode 2xe2x80x9d processes taught there. In mode 1, all the distances to be determined must be fully contained in each camera view. In mode 2, a distance can be determined which is too large to be contained within any single camera view.
A general measurement reference apparatus comprises a reference target array which is placed near to and fixed with respect to an object of interest inside an enclosure by means of a reference array holding apparatus and a reference array insertion apparatus. This measurement reference apparatus can be used to make either perspective dimensional measurements as I teach or to make conventional photogrammetric measurements.
A general measurement apparatus to be used with the new methods comprises a reference target array which is placed near to and fixed with respect to an object of interest inside an enclosure by means of a reference array holding apparatus and a reference array insertion apparatus. This apparatus allows for the possibility of using the reference array holding apparatus to attach the reference array directly to the object of interest. In this case, the reference array and the array holding apparatus are inserted into the enclosure by the insertion apparatus. The general measurement apparatus also comprises an endoscopic camera located inside the enclosure. The camera is moved with respect to the object by a camera moving apparatus. Images formed by the camera are measured with an image measurement apparatus. These measurements are supplied to a computing apparatus which computes the desired three-dimensional distances. The measurement results are displayed to the user by a display apparatus.
In a first specific embodiment of a measurement apparatus, the camera is a flexible endoscope, and the camera moving apparatus can either be the internal articulation provided within the endoscope or the endoscope can be subjected to any other motion which meets a specific requirement that is explicitly taught.
In a second specific embodiment, the camera is a substantially side-looking rigid borescope, and the camera is moved by translating the borescope in a direction substantially along its length. A first borescope holder is taught that provides the user with a convenient way to hold the borescope steady at each viewing position in this embodiment.
In a third specific embodiment, the camera is a rigid borescope (either forward-looking or side-looking), and the camera is moved by rotating the borescope about an axis of rotation that is substantially perpendicular to the length of the borescope and that is nearly coincident with the inspection port through which the borescope has been introduced to the interior of the enclosure. A second borescope holder is taught that provides the user with a convenient way to hold the borescope steady at each viewing position in this embodiment.
Any of these specific embodiments of measurement apparatus can be used with any of a number of specific implementations of reference target arrays and reference target array insertion and holding apparatuses. The array of reference targets can be introduced into an enclosure, brought to a position near the object of interest, and held at a fixed position with respect to the object using a variety of modifications of standard endoscopic inspection accessories. In specific implementations, the array of reference targets is supported by a rigid, a semi-rigid, or a flexible endoscope guide tube. Optionally, the array may be supported by a wire which is fed through a lumen in the wall of the guide tube. In another implementation, the array of reference targets is supported by a second endoscope. In additional implementations, the reference target array is supported by a support wire which is mounted to the enclosure at an inspection port.
Arrays of reference targets may be marked on rigid or flexible substrates. In certain preferred implementations, the substrate of the array is planar, while in others, the array is three-dimensional. For planar arrays, a feature is added to the array that makes it easy for the user to determine an additional piece of information that is necessary if one wants to minimize the number of reference points used in the measurement. A number of specific three-dimensional arrays are taught that have the advantage that they can be characterized using only one- and two-dimensional metrology.
Optionally, the substrate of the array may contain one or more apertures thus enabling one to view the object through the array. In another implementation, the array is marked on a loop of wire. In additional implementations, the array may have a variable orientation with respect to its support member. In a specific implementation, an insertion tool is provided that enables a flexible reference target array to be collapsed for insertion through an inspection port and then expanded once it is inside an enclosure. Other specific implementations provide a combined reference array, support means, and insertion means, where the flexible reference target array is mounted at the distal end of the support means, and where the array can be collapsed to a small size for insertion, and expanded to full size for measurement.